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ABSTRACT 

Engineering students benefit from an active and interactive classroom environment where they 
can be guided through the problem solving process. Typically faculty members spend class time 
presenting the technical content required to solve problems, leaving students to apply this knowledge 
and problem solve on their own at home. There has recently been a surge of the flipped, or inverted, 
classroom where the technical content is delivered via online videos before class. Students then 
come to class prepared to actively apply this knowledge to solve problems or do other activities. In 
this paper, recommendations are made for applying this educational technique to large engineering 
classrooms. These recommendations are based both on a literature review of flipped classes and 
the evaluation of a case study of a large Introduction to Environmental Engineering class. The case 
study evolved from a traditional lecture-based classroom through two different versions of a flipped 
classroom. Evaluation of students’ interaction, preferences and performance are used to make rec¬ 
ommendations about the video time, use of class time, course organization and student assessment. 

Key Words: classroom flip, inverted classroom, active learning 


INTRODUCTION 

Instructional environments that allow for students to be more actively engaged with course 
material are more likely to lead to greater learning gains. The literature in engineering and science 
education continues to encourage faculty and instructors to use class exercises that require students 
to be actively engaged in the course material, as opposed to being passive recipients of information. 
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Chi (2009) demonstrated that student learning increases as engagement in the material shifts from 
passive to active, in which students physically engage somehow in the activity, and from active to 
constructive, in which students are required to produce new outputs that go beyond information 
presented in class. In his study of a large sample of physics students, Hake (1998) found that students 
who were in classes that incorporated interactive-engagement methods made significantly larger 
gains on a concept inventory as compared to those who were enrolled in traditional courses, sug¬ 
gesting that the instructional method impacted students’ problem solving abilities. Prince (2004) 
conducted a literature review of the benefits of active learning and found that a variety of different 
active learning types do result in increased student learning, although some variability in effect 
sizes was evident. 

Entwistle and Peterson (2004) provide a conceptual framework of the influences on student 
learning. One of the many factors that can influence student learning concerns the teaching-learning 
environment, which includes both how course content is organized and presented and how the 
classroom environment is designed. In an early paper, Entwistle (2000) notes that teacher-focused, 
content-oriented approaches to teaching in which goals are related to information transmission are 
more likely to result in surface learning approaches to studying and limited understanding. In con¬ 
trast, student-focused orientations in which facilitation of understanding is a primary goal is more 
likely to lead to deeper approaches to studying and a more thorough understanding of the material. 

While many instructors understand that these methods may be beneficial to students, some may 
be reluctant to use instructional methods that require students to engage in active or construc¬ 
tive learning in their courses. As Michael (2007) found, one of the primary challenges that faculty 
perceive is, “Active learning takes too much class time and coverage of content will suffer” (p. 43). 

In the classroom flip, the activities traditionally done in class and at home are flipped, or inverted. 
Instead of watching the lecture in class, students are asked to complete necessary preparatory work 
outside of class, often times in the form of an online, virtual lecture that has been prepared by the 
instructor. Class time is then freed up so that the instructor can guide students to complete active, 
constructive, or interactive activities during regularly scheduled class time (Baker, 2000; Lage & 
Platt, 2000; Lage, Platt & Treglia, 2000). 

The purpose of this paper is to make evidence-based recommendations to faculty interested in 
teaching a flipped course. These recommendations are based both on data from the literature and 
empirical evidence from a case study presented here. First, the paper will describe the classroom 
flip method and provide a literature review on the current use of the classroom flip technique in 
engineering education. Furthermore, a case study of the implementation of the classroom flip in a 
large introductory environmental engineering course will be discussed. This course evolved from a 
lecture-based course through two different versions of a flipped course. Changes between the two 
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versions included shortening the length of the video content and changing the classroom activities. 
A description of each implementation strategy is provided, including information on what students 
are required to do outside of class, what activities are done during class time, and class assessment 
strategies that encouraged students to be prepared. Evaluation of the classroom flip was conducted 
in the form of post-course surveys and focus groups. Based on the evaluation data of both versions 
of the flipped class, specific strategies are offered to instructors who are interested in pursuing the 
instructional method in their own courses. 


BACKGROUND AND LITERATURE REVIEW 


Description of the Classroom Flip 

In the traditional lecture class, the instructor acts as the information provider and as the lead 
of the class, sometimes termed the “sage on the stage” (King, 1993). In the most passive version 
of these classrooms, students do not interact with the instructor or with each other. They serve as 
recipients of information with little or no engagement with course material. In the traditional class¬ 
room, information primarily passes from the instructor to the students as diagrammed in the left side 
of Figure 1. While active learning techniques can be implemented in an interactive lecture format, 
such as think-pair-share activities in which students interact with their peers, some instructors have 
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resistance to completing these tasks during class time as it reduces the amount of time in which 
new material can be introduced to students. 

A diagram of the flipped classroom is available on the right side of Figure 1. In the flipped class, 
the students gain technical knowledge through online videos and complete an online assessment, 
which ensures that students are sufficiently prepared to participate in the in-class activities. These 
in-class activities may include problem solving, discussions, brainstorming, design work, guest 
speakers or field trips. In most engineering classes, problem solving may be done in class, which 
normally would be assigned as homework. Students work with one another in groups to complete 
the activities. Instructors act as a guide while students complete the in-class activities, providing 
assistance as needed to student groups. If many students are having similar problems, the instruc¬ 
tor can direct the attention of the whole class to discuss content relating to the common stumbling 
block, in a form of just-in-time teaching (Novak, Gavrin, Wolfgang, & Patterson, 1999). In all cases the 
instructor becomes a guide to students, rather than the “sage on the stage”. In the words of Baker 
(2000), who wrote one of the earliest papers on the method, the instructor acts as the “guide on 
the side.” Table 1 provides a comparison of different course characteristics of the traditional lecture 
and flipped classes. 

The flipped class format has several benefits to both students and instructors. Shifting the lecture 
to out-of-class time allows the instructor to still “cover” important material while having time for 
students to engage in active learning exercises in class. For students, the shift in instruction from 
passive to more active provides an opportunity to engage more deeply in the class material. The 
flipped classroom also allows for more faculty-student and student-student interaction during class 
time. This can potentially provide students with greater opportunities to integrate higher order 
cognitive skills in the classroom, focusing less on remembering and memorizing material and more 
on applying, analyzing, and potentially creating (Anderson & Krathwohl, 2001). The classroom flip 


Course characteristics 

Traditional Lecture 

Flipped 

Role of instructor during class 

Information provider 

Guide 

Role of student during class 

Information recipient 

Active participant in class 

Out-of-class activities 

Solving problems, reading 
textbook, projects, preparing 
for quizzes/exams 

Watch online lecture or read assigned material (before 
class), complete problem sets & preparing for quizzes/ 
exams (after class) 

In-class activities 

Instructor-led lecture 

Varies (i.e. problem solving, projects, discussion, 
brainstorming, field trips) 

Role of assessment 

Primarily summative in 
nature 

Formative “gate checks” to ensure preparedness; Both 
formative and summative assessment 


Table 7. Comparison of traditional lecture and inverted classroom in engineering. 
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provides a structure for students’ out-of-class time, guiding students on where their efforts should 
be directed. This is a skill that is troublesome for some students who may struggle with determining 
what is most important when assigned readings from a textbook. Watching online videos allows 
students to obtain the technical content in their own time and at their own pace. They can re-watch 
portions that were unclear to them or prepare for later course assessments. Most importantly, stu¬ 
dents receive help from the instructor during class time during critical periods of learning. 

Prior work on the classroom flip in engineering education 

Some of the earlier work on the classroom flip was done by Lage and colleagues (2000) who 
coined the phrase “inverted classroom.” Since then, the method has begun to proliferate in both K-12 
school settings (e.g. Fulton, 2012; Bergmann & Sams, 2012) and in higher education. The classroom 
flip instructional method appears to be growing in popularity in engineering education, given both 
the number of conference papers dedicated to the method as well as the proliferation of informal 
discussions by faculty at these conferences. 

The first paper in the engineering education literature that discussed the classroom flip was by 
Bland in 2006, who presented at the annual meeting of the American Society for Engineering Edu¬ 
cation (ASEE). The instructor used a low-tech version of the method to assign pre-class activities 
and assignments in an electrical engineering course (Bland, 2006). Gannod and colleagues followed 
this study by implementing the flip in a software engineering class (Gannod, 2007; Gannod, Burge, 
& Helmick, 2008). 

In 2009, the use of the classroom flip instructional method began to proliferate in engineering 
education, with several conference papers published. Zappe, Leicht and colleagues describe how a 
portion of an architectural engineering class was flipped (Zappe, Leicht, Messner, & Litzinger, 2009; 
Leicht, Zappe, Messner, & Litzinger, 2012). They collected data on how students used video lectures 
and students’ perceptions of the instructional method. Their primary suggestions on the method 
included: 1) require an online quiz to ensure students come to class prepared, 2) keep the videos 
fairly short in order to encourage students to watch them, 3) briefly review course content to make 
sure students are prepared, and 4) consider adding multi-media to online lectures to increase stu¬ 
dent interest. An examination of the impact of the classroom flip on learning gains was not found 
to be significant across different semesters with the same instructor. 

In a series of papers starting in 2009, Dollar, Steif, and Ulseth describe efforts to implement the 
classroom flip in a statics course (Dollar & Steif, 2009; Dollar, Useth, & Steif, 2011; Steif & Dollar, 
2012). The authors used an online system consisting of simulations and other interactive activities 
to teach students statics. Papadopoulos and colleagues (Papadopoulos, & Santiago-Roman, 2010; 
Papadopoulos, Santiago-Roman, & Portela, 2010) followed the efforts of Dollar and Steif and 
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implemented similar strategies in statics courses at the University of Puerto Rico using the same 
online learning environment. 

In 2009, Toto and Nguyen worked with a faculty member from industrial engineering to flip por¬ 
tions of a junior level course. The authors found that students had a positive perception to using 
class time for problem-solving and hands-on activities, but admittedly preferred face-to-face lectures 
as compared to virtual lectures. Kellogg (2009) describes another variation of the classroom flip, 
in an industrial engineering course on cost estimation in which the students were required to read 
online texts and complete embedded activities prior to coming to class. 

Demetry (2010) describes the evolution of her use of the classroom flip in a materials science 
course. In her revised version of the course, Demetry used online “microlectures” followed by a 
short readiness assessment test. During class time, Demetry asked students to work in teams to 
complete a problem set that required higher level cognitive skills. The role of the teacher was to 
provide assistance and whole-class clarification as needed. Although Demetry collected assessment 
data, because the paper was a Work in Progress, no information on the project impact was provided. 

The 2012 conference of ASEE had several papers dedicated to the classroom flip. Laman, Brannon, 
and Mena (2012) presented a paper on a low-tech version of the classroom flip in which they re¬ 
quired students to complete a preparation reading and a pre-class assignment concerning a design 
problem. The instructor did not flip the entire course, but used a scaffolding strategy in which stu¬ 
dents had increasing responsibility for being prepared as the semester progressed. Talbert (2012) 
used the inverted classroom in an introductory MATLAB course at a liberal arts college. Students 
were required to watch either MATLAB tutorials or instructor-created screencasts. Instead of us¬ 
ing online quizzes to assess preparedness, the instructor required students to complete a quiz via 
classroom response systems (clickers) at the start of class. Thomas and Philpot (2012) compared 
student grades in sections of a mechanics of materials course that were inverted and traditionally 
taught. No significant differences were found in exam scores between the two formats, although 
a threat to the validity of the study is that different instructors taught the sections. At a regional 
ASEE conference in 2012, Warter-Perez and Dong describe their efforts at flipping a portion of 
their course. They required students to engage in pre-project activities outside of class and then 
used class time for a variety of purposes, including interactive problem solving and inquiry-based 
activities. 

Table 2 provides a listing of the papers found describing or using the classroom flip method 
in engineering education. While the number of conference papers in engineering education 
about the classroom flip seems to be increasing, as of this paper submission, there have been 
no works in peer-reviewed journal articles in engineering education describing the method 
and its impact. 
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Author and year 

Discipline 

Sessions 

flipped 

Out-of-Class Activities 

In-Class Activities 

Assessment of 
Preparation 

Bland (2006) 

Electrical 

engineering 

Whole 

course 

Pre-class activities and 
assignments 

Problems, case studies, 
physical examples, 
group-work 

Assignments 

Demetry (2010) 

Materials 

Science 

Whole 

course 

Online microlectures 

Group problem solving 

Readiness 

assessment tests 

Dollar &Steif (2010); 
Dollar, et al., (2011); 
Steif & Dollar (2012); 
Papadopoulos & 
Santiago-Roman 
(2010); 

Papadopoulos, et al. 
(2010) 

Statics 

Whole 

course 

Online modules and 
simulations 

Concept questions 
discussed with peers; 
active learning exercises 

“Learning by 

Doing” and 
“Did I Get This” 
Exercises; Student 
perception surveys 

Gannod (2007); 
Gannod, et al. (2008) 

Software 

engineering 

Whole 

course 

Podcasts (Powerpoint 
with voiceovers) 

Programming projects 

Not described 

Kellogg (2009) 

Industrial 

engineering 

Course 

segment 

Online embedded 
interactions; online text 

Active/collaborative 
problem solving in class 

Online quizzes 

Laman, et al. (2012) 

Civil 

engineering 

Course 

segment 

Pre-class preparation 
readings and design 
problems 

Collaborative work on 
design problems 

Online evaluations 
of reading 
comprehension 

Talbert (2012) 

MATLAB 

Whole 

course 

Mathworks videos; 
Instructor-created 

screencasts 

MATLAB assignments 

Quizzing via 
clickers at start of 
class 

Thomas & Philpot 
(2012) 

Mechanics 
of materials 

Whole 

course 

Readings, animations, 
and videos 

“Homework” either 
individually or in 
groups 

Not described 

Toto & Nguyen (2009) 

Industrial 

engineering 

Course 

segment 

Online lecture 

Problem solving and 
hands-on activities 

Online quizzes 

Warter-Perez & Dong 
(2012) 

Digital 

Engineering 

Course 

segment 

Pre-project assignments 

Problem solving, 
inquiry-based activities, 
interactive lectures, 
collaborative projects 

Assignments 

Zappe, et. al (2009); 
Leicht, et. al (2012) 

Architectural 

engineering 

Course 

segment 

Narrated screencasts 
using Tablet PC 

Work on group projects 

Online quizzes 


Table 2. Flipped courses in the engineering education literature 


No studies have yet been determined as to why the method is attractive to faculty or why the 
method seems to be growing in popularity. However, there are some hypotheses. One primary 
reason is that modern technology has made it fairly easy to capture a lecture. In the engineering 
education literature, the earliest innovators used podcasts (i.e. Bland, 2006; Gannod, 2007). More 
recent adopters of the method are using online simulations and narrated screencasts (i.e. Zappe, 
et al., 2009; Talbert, 2012). Software programs for creating screencasts abound and include both 
free versions such as Jing and sophisticated tools for purchase such as Camtasia. Tablet PCs also 
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provide instructors with the opportunity to communicate through writing and problem solving. 
Faculty can easily upload videos to university-owned, account-controlled streaming servers or 
publicly to sites such as YouTube. 

A second potential reason is the increase in the amount of online information that is readily 
available to students. Using Google, students can access a plethora of information immediately. If 
information is so readily available at the push of the button, some instructors may question whether 
or not their role as “information givers” is no longer needed. With the advent of massive open 
online courses (MOOCs), perhaps some instructors may wonder what the future may hold for the 
traditional face-to-face classroom. A possible model for the classroom flip is using a MOOC as the 
material that students complete outside of class and then using class-time to complete activities 
or projects relating to the MOOC. 

A third reason is that many instructors are being asked to teach online courses. Some of these 
instructors create video content that is intended to be used for streaming content in an online 
course environment. Given that these videos are already created, instructors have an easy entry 
into flipping their face-to-face courses. This latter reason is the primary impetus for flipping the 
course described in this paper. 


CASE STUDY 


The instructor had previously taught a large (80 - 90 students) Introduction to Environmental 
Engineering course in a traditional lecture format for two semesters before deciding to flip it. In the 
traditional offering, she used the class time to present lecture material and included some active 
learning components such as brainstorming and short problem solving in the classroom. Outlines 
of lecture notes were provided to the students; the instructor filled in the notes using a Tablet PC 
which was projected to the class. 

Students’ use of office hours motivated her to flip the course in order to increase active learning. 
During these office hours, an average of 8 -10 students would come with questions regarding the 
homework problems. Although the instructor had covered this material in detail in the class lecture, 
students still had many questions. One possible reason for these questions is that students did not 
need the information covered in class until the homework assignment was due. Once the students in 
office hours asked questions directly applied to the problem set, they seemed to better understand 
the material and were then able to complete the problems successfully. In addition the instructor 
noticed that the students were teaching each other. This benefitted both the student acting as 
“teacher” and the student acting as the “learner.” The idea of flipping the classroom appealed to 
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the instructor in order to mimic this student-centered learning in the classroom for all students, not 
just those who attended office hours. In addition, she hoped to increase both faculty-student and 
student-student interaction during class time, not just during office hours. 

Figure 2 shows the progression of teaching of this course through two different versions of 
flipping, including changing the format of the online videos and the type of activities completed 
during class time. During Version 1 of the flipped course, the instructor observed that the students 
were indeed more actively involved during class time but she also observed some challenges. The 
students were concerned with the amount of time they were spending on the class. Because they 
had to come to class once per week and still watch 3 lectures online, they expressed a desire to 



Format 

At home 
Before class 

Monday & 
Wednesday 

Friday 

At home 
After or 
during week 

Assessment 

Traditional 


Lecture in class 

Homework 

Study for 
Quizzes/Exams 

3 Exams 

Weeks later (F08, F09) 

6 Quizzes 

1 - 2 Weeks later (SI2) 
Cumulative Final 

Flipped 
Version 1 
(VI) 

Watch Full 
video lectures 

Take On-line 

assessments 

Smaller groups 

Once per week (M,W or F) 

Review material 

Work on extra problems 

Field trips 

Homework 

Study for 
Exams 

3 Exams 

Weeks later 

Cumulative Final 

Flipped 
Version 2 
(V2) 

Watch Short 
video segments 

Take On-line 

assessments 

Review material 

Work on 
homework 
problems 

Field Trips 

Quiz 

Homework 
(if necessary) 

Study for 
Quizzes 

10 Quizzes 

same week 

Cumulative Final 

Figure 2. Comparison of a Flipped Class with a Traditional Class taught by the same 

instructor. In each flipped format , the students watched online video lectures before 

coming to class and worked on problem sets during class time. In VI , the online lectures 

were created by recording the lecture during class time while in V2 the lecture material 

is presented in shorter video segments. Also in VI, the class time was used for additional 

problem sets white in V2 the class time was used to work on homework problems. Finally in 

V2, the student learning was assessed at the end of each week via an in-class quiz while in Vi 

the learning was assessed via two mid-term exams. The shading represents time the students 

spent in class with the instructor. 
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earn 4 credits for the course instead of 3. In addition, the students were required to come to class 
and work on class problems in addition to the homework problems. This resulted in students being 
present in class but not actively involved in the class discussion and problem solving. 

In Version 2, the time of the lecture material was reduced, although the content remained the 
same, based on recommendations from prior studies (Zappe, Leicht, Messner, & Litzinger, 2009; 
Leicht, Zappe, Messner, & Litzinger, 2012). In addition, the class time was used for the homework 
due at the end of the week and students were invited (but not required) to come to class every day. 
With the revisions in Version 2, the students did not comment negatively about the time commit¬ 
ment and were more engaged in problem solving in class since they knew they were getting help 
on the “homework” problems. Details of the lectures and class time are given below. 


COURSE DESCRIPTION 


Use of Videos 

All the videos in both versions were created with assistance from the Continuing and Distance 
Education Office, located in the College of Engineering at Penn State. Outline notes were created 
using Word and these notes were converted into Portable Document Format (PDF) documents. 
During the lectures the instructor filled in the notes using PDF Annotator®. The videos were housed 
on a streaming server in the College of Engineering and provided to students through links in the 
course management site (ANGEL). 

Videos created during class time (Version 1): During Fall 2010, the instructor’s videos were re¬ 
corded during live lectures. Notes were filled in using a Tablet PC. Forty, 50-minute lectures were 
created for a total of 33 hours of video. These videos included student questions, class administra¬ 
tion (reminders of due dates, etc.), exam reviews and brainstorming time. Each week students were 
required to watch three of the 50-minute videos. 

Short video segments (Version 2): For Version 2 of the flip, the lecture material was reorganized 
into 11 self-contained modules. These modules were recorded in a recording studio without a student 
audience the semester before the course began. The videos were created with a Wacom interactive 
pen display screen and video capture. Outline notes were created using PDF Annotator software. 
These videos contained contextual stories, derivations ad sample example problems. Each week, 
students were required to watch one module consisting of multiple video segments. The module 
lengths varied from 87 minutes to 143 minutes for a total of 21.7 hours of video created. Each module 
contained 7 to 18 short video segments (for a total of 126 video segments). These segments varied 
from a few minutes to up to 20 minutes in length. Although the goal was to create videos no longer 
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than 10 minutes, which is consistent with the prior literature (Zappe, et al., 2009), the segments were 
created to the length necessary to communicate information relating to the primary objectives of 
that segment. The short video segments were titled appropriately, allowing students to selectively 
re-watch segments of interest. The length of each video was also presented to the students so they 
could budget their time effectively. 

The preparation of these videos took approximately 30 hours to create in total. One module was 
re-recorded after the first semester to improve content. Although creation of the videos took more 
time upfront, the lecture-prep time was saved in subsequent semesters. 

In class time 

During each version of the flipped course, the class met in person three times a week for 50 min¬ 
utes each. In both versions of the flipped course, the instructor used a short amount of time (5 - 15 
minutes) at the start of class to address students’ concerns and questions from the online lecture 
material. These concerns and questions were ascertained from the weekly online assessment (see 
online assessment “gate check” below). 

In addition, students in both versions of the flipped course were able to attend field trips. These 
trips were taken to locations at or near campus and included tours of the recycling facilities on 
campus, a wastewater treatment plant, and a LEED-certified building. These field trips were previ¬ 
ously prohibitive because of the size of the class. By flipping the class the instructor could take one 
third of the students at a time on the field trip. In a normal class that would mean that the students 
would miss a whole week of course content. In the flipped class the students don’t miss technical 
content since they can obtain that content from the videos. 

Version 1: During the first version of the flip, the instructor divided the large class into three smaller 
groups. Each group was assigned only one day of the week to attend class. The purpose of this was 
to mimic a small classroom experience. Attendance, which counted for 10% of their grade, was re¬ 
corded for each group on that day. The students were given additional problem sets to complete in 
class. These problem sets were not handed in and not graded. Although these problems were similar 
to the problems assigned for homework, the in-class problems were an additional activity that the 
students did during class. The students still had to complete their regular homework assignments 
outside of class. During Fall 2011, the class time was used for the following: 

1. Problem solving (8 class periods). Students came to class once per week for a total of 8 
problem solving sessions. (The instructor attended 24 class periods in total as each class pe¬ 
riod was repeated three times to accommodate the three smaller groups.) After addressing 
students’ questions and concerns, the instructor took attendance and then handed out class 
problems. The students would work on these problems during class individually or in groups. 
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The instructor would sometimes solve parts of the problem on the board. As mentioned above, 
these problems were assigned in addition to homework. 

2. Field trips (3 class periods). Students went on three field trips in smaller groups. 

3. Review sessions (3 class periods). Each student attended one review session before each of 
the three exams. 

4.Student presentations (1 class period). Students could choose to prepare an extra credit pre¬ 
sentation on a topic of their choosing. 

Version 2: In Version 2, the in-class time on Monday and Wednesday was devoted mostly to solv¬ 
ing the homework problems due on Friday. Attendance was not recorded. A quiz was given each 
Friday. During Fall 2012, class time was used for the following: 

1. Solving “homework” problems (20 class periods). The instructor would open the class by 
addressing students’ concerns and questions from the online assessment. Then the students 
would work on the homework problems assigned for the week. The instructor and teaching 
intern would walk around the room to answer questions. Students were encouraged, but not 
required, to work in groups. If common questions were asked, the instructor would address 
the question at the board for all students. 

2. Quizzes (10 class periods). The students would take an in-class quiz on 10 Fridays during the 
semester. 

3. Field trips (4 class periods). Students were divided into 2 or 3 groups and went on 4 total field 
trips. In Version 2, two trips to different wastewater treatment plants were made. One of these 
was on campus; the other was located in the township. 

4. Guest speakers (2 class periods). There were two guest speakers that discussed Marcellus Shale 
drilling and indoor air quality. 

Class Assessment 

Online assessment (gate check) 

In both versions of this flipped course, students would take a short (12 - 18 question) online assess¬ 
ment after watching the videos but before coming to class. This occurred on Sunday evening before 
the Monday class. This online assessment served as a “gate” for students to check for understanding 
and to hold them accountable for watching the videos. Most of the questions on the online assess¬ 
ments were multiple-choice and addressed qualitative concepts and/or required simple calculations. 
On each online assessment, the students were also asked to share the clearest and muddiest parts 
of the video lecture. This question was open-ended and counted in their final assessment grade. The 
instructor addressed the “muddy” points in class. In both versions of the flipped course, the online as¬ 
sessments were worth 10% of the grade. In addition, the lowest online assessment grade was dropped. 
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In-class Assessment 

In Version 1, students completed two midterms exams and one cumulative final exam. The mid¬ 
term exams were given in the evening, not during class time and were each worth 20% of the final 
grade. The Final Exam was worth 20% of the final grade. 

In Version 2, the students did not take the mid-term exams but did have a cumulative final. The in¬ 
structor decided to have the students take shorter in-class quizzes corresponding to each module. There 
was one exception of a quiz that covered two modules. These quizzes replaced semester exams and 
were worth 50% of the total grade. Similar to Version 1, the final exam was worth 20% of the final grade. 

Evaluation Methods 

Previous studies on flipping have indicated many advantages including active learning in the class¬ 
room for the students and a move for the instructor from a “sage on the stage” to a “guide on the side”. 
However, many faculty members will be reluctant to incorporate flipping in their classroom if it creates 
significant student resistance. It was our aim in this work to assess students’ perceptions and performance 
during two different versions of the flipped classroom over several semesters so that we could make 
specific recommendations to faculty interested in flipping their engineering courses. We specifically 
hypothesized that the more customized and shorter videos and the use of in-class time to complete 
homework rather than additional problem sets would result in more positive feedback from students. 

In order to answer this hypothesis and make recommendations to faculty, several guiding questions 
were used to understand the use of the classroom flip and to understand the differences between 
the two versions of the flipped class. These questions follow: 

7. How do students interact with both on-line lectures and flipped classroom time? 

For this research question, students’ behaviors with online lectures were explored, including how 
many of the videos they report watching, how often they review the videos, and reasons why they 
reviewed the videos. In addition, because faculty may be concerned about the potential increase in 
number of absences, students were also asked how often they were absent. 

2. What are students’ preferences for on-line content delivery and use of class time? 

For this question, students are asked about their preferences regarding length of the online videos, 
their preference for coming to class for a lecture versus watching a lecture online, and the perceived 
benefits of using class time for problem solving. Comparisons are made between the format of Version 
1 and Version 2 to better understand which model is preferred by students. 

J. How does the type of instruction affect student learning as measured by cumulative final exam scores? 

One of the most important questions asked when an educational innovation is implemented is 
whether student learning is impacted. In order to understand impact on student learning, final exam 
scores were compared for six semesters in which the instructor had taught the course. The instructor 
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taught traditionally using lecture for three of these six semesters. During the other three semesters, 
the instructor taught using Version 1 of the flipped course twice and Version 2 of the flipped course 
once. (The instructor taught the course one additional time in Fall of 2010. However, because of dif¬ 
ferent grading policies during this semester, final exam score data was not included in this analysis.) 
During each semester, the students were administered a two-hour cumulative final exam. The content 
was not exactly the same but care was taken to ensure that the questions were as similar as possible. 

4. What do the students find most valuable about the flipped course? 

For this section, students’ preferences for the flipped format versus traditionally-taught courses 
are discussed, using results from both surveys and focus groups. Additionally, students’ reasons for 
their preferences were collected. 

The evaluation of the classroom flip was conducted using a combination of qualitative and quantitative 
measures. The quantitative measures included post-course surveys that focused on students’ prefer¬ 
ences for the flipped versus traditional instruction, their behaviors regarding video watching and class 
attendance, and perceived impact on interest and learning. The format of the items varied, including both 
closed and open-ended questions. Data from the open-ended questions was coded to find common 
themes in students’ responses. The survey was developed by education experts located in the College 
of Engineering’s teaching and learning center. The Leonhard Center for the Enhancement of Engineering 
Education. The instruments were customized for understanding the impact of the specific course situation. 
Because the survey was primarily constructed with individual, stand-alone items of students’ behaviors 
and preferences rather than inclusion of summated rating scales, no psychometric analyses (such as an 
item analysis or reliability analysis) would be appropriate to conduct. However, the items were reviewed 
internally by multiple researchers in order to ensure that items were functioning as intended. A copy of 
the survey for Version 1 and Version 2 are available in Appendices A and B. 

The surveys were administered online using a commercial surveying software tool called Qual- 
trics. Participants in the evaluation were first asked to complete an informed consent document, in 
accordance with Penn State’s Institutional Review Board. Out of 59 enrolled students, 36 students 
(61% of students) filled out the post-surveys for Versionl while 66 out of 86 (77% of students) filled 
out the post-survey for Version 2. Participants received 1% extra credit on their final grade as an 
incentive for completing the surveys. 

Descriptive data from the post-course surveys is presented for the first two research questions. In 
addition, for certain questions, comparisons are made between students’ perceptions during Version 
1 and Version 2 of the course. These comparisons will help to determine if the changes made during 
Version 2 result in more positive student perceptions. 

In addition, during Version 2, focus groups were conducted to gather additional information on stu¬ 
dents’ perceptions of the instructional technique. Eight students attended each of three focus groups. 
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In order to encourage participations, students were offered additional extra credit for attendance. The 
students attended the focus group for approximately 45 minutes. A semi-structured format was used to 
initiate discussion and assess the students’ opinions regarding the flipped format of the course. Ques¬ 
tions concerned their perceived value of the course and the instructional methods. The focus group 
protocol was devised by the research team, including an expert in educational assessment. All sessions 
were audio recorded and summarized for later analysis. As an additional source of evaluation data, dur¬ 
ing Spring 2013, four students were identified who enrolled in two separate environmental engineering 
courses with the instructor. One of the courses was taught in Version 2 of the classroom flip and one 
of the course was taught with a lecture format. These four students participated in a one-hour focus 
group to discuss their perceptions of the classroom flip and which instructional method they preferred, 
given the same instructor. Copies of the focus group protocols are available in Appendices C and D. 


EVALUATION RESULTS 

How do students interact with both on-line lectures and flipped classroom time? 

The majority of students reported watching the online video content. In both versions of the 
flipped class, the majority of the students watched almost all the videos. Figure 3 shows that 53% of 
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Figure 3. Number of modules and video segments watched by students. In Version 1 (a), 
there were 40 videos and 53% students reported watching all 40 videos white total of 75% of 
students reporting watching between 33 and 40 of the lecture videos. In Version 2 (b), 45% 
of the students reported watching all 132 video modules while 78% reported missing less 
than 10 out of 132 video segments. 
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the students watched all 40 videos in Version 1 while 45% watched all 132 video modules in Version 
2. Although it is difficult to compare directly since there are not the same number of videos in each 
Version, 17% of the students in Version 1 reported missing 60% of the content (watching less than 
24 of the 40 videos) while in Version 2 only 4% of the students reported missing 76% of the videos 
(watching less than 101 of the 132 videos). This data suggests that regardless of video format, whether 
they are longer or shorter in length, students will generally watch the videos in the flipped class. 

One of the main advantages of online lectures for students is the ability to re-watch portions of 
the videos. Figure 4 shows that the likelihood of re-watching the video lectures is about the same 
for both versions. Approximately 10% of the students reported re-watching the lectures all the time 


50% 



Figure 4. Students’ responses to the question: “Did you review portions of the online 
modules that seemed unclear to you?” for Version 7 (gray) and Version 2 (black). Over 65% 
of the students reported reviewing unclear portions of the video at least occasionally in both 
versions of the course offered. 
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Reason for re-watching 

% 

reported 

To clear up a misunderstanding 

70% 

For the online assessments 

70% 

Help with HW 

67% 

Because I was distracted the first time I watched it. 

55% 

Review for quiz in class 

47% 


Table 3. Reasons students reported re-watching the video lectures during Version 2. 


while 11% (in Version 1) and 6% (in Version 2) report never re-watching the videos. 

During the evaluation of Version 2, we specifically asked students why they re-watched video seg¬ 
ments. They were able to select more than one response. As shown in Table 3, 70% of the students 
stated that they re-watched to clear up a misunderstanding and to work on the online assessment 
while 67% percent reviewed the videos in order to receive help with homework problems. In addition 
over half (55%) of the students re-watch the lectures because they lost focus the first time around 
while 47% re-watch lectures to help study for the in-class quiz. 

During Version 1, the students were required to come to class once a week and attendance was 
taken. However during Version 2, the students could choose to come to class two days a week to 
review the material (attendance was not recorded) and work on problem sets. During the evaluation 
of Version 2, we asked students about their classroom attendance. Figure 5 shows that 24% reported 
coming to all the class periods and 50% reported missing two or less class periods. In addition, 62% 
said they missed class as many times in this course compared with their other courses while 9% said 
that they were absent fewer times as compared to other courses. Reasons for not coming to class 
included: being able to finish the problems on their own (55%) or preferring to work on problems on 
their own (33%). They also reported having other academic priorities (34%) and that they “didn’t feel 
like coming” (25%). Although we did not take attendance every day during Version 2, the students’ 
reported absences are consistent with the instructor’s general observations during the semester. 

What are students’ preferences for online content delivery and use of class time? 

An area of interest was students’ preferences in terms of the length of each video segment. Is it 
better to have the lecture broken up into smaller videos, even though the students have to manually 
start each video segment? During Version 2, we asked the students about their preferences regard¬ 
ing the length of videos, by asking how they would divide 120 minutes of online lectures. Students 
report preferring shorter mini-lectures instead of one long lecture. Figure 6 shows that 62% of stu- 
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0 1-2 3-5 4-9 10-15 16-20 

Number of class times missed 


Figure 5. Number of problem solving class times students reported missing during 
Version 2 of this course. Attendance was not recorded for these class times. There were a 
total of 20 problem solving class times (in addition to 10 quiz class times, 8 class times for 
field trips and 2 class times for guest speakers). 


dents would prefer twelve 10-minute videos while 24% would prefer six videos at 20 minute each. 

How do the student preferences change with the type of online delivery? Recall that in Version 
1, the videos were recorded from a live class; each video was 50 minutes in length. In Version 2, the 
videos were much shorter in length, although a greater number of videos were provided. 

Students in both versions were asked to rate their level of agreement with the several statements 
using a Likert-type scale from Strongly Disagree (coded as 1) to Strongly Agree (coded as 5). Average 
scores were calculated for each item. The results are given in Table 4. First, students were asked their 
level of agreement with the following statement: “I would prefer to use class time for problem solving 
activities, rather than listening to a lecture.” Students in both sections seemed to find this helpful. There 
was no significant difference in the average rating for this item between versions [f(99) = 1.17, p = 0.24]. 

Students were asked about the benefits of moving lecture material out of class with the ques¬ 
tion: “I feel that moving lecture material out of class and having more time to work during class is a 
beneficial use of course time.” Students in Version 2 rated this item significantly higher [f(98) = 2.65, 
p = 0.0093]. This suggests that students in Version 2 found working in class to be more beneficial 
than those in Version 1. Recall that in Version 1 the students were only required to come to class once 
per week and used that time to work on additional problem sets. In Version 2, the students were not 
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4 @ 30 min 6 @ 20 min 12 @ 10 min 24 @> 5 min 

Preference of time for length of on-line lectures 

Figure 6. During Version 2, students were asked to choose how to watch 120 minutes of 
video. 62% of the students choose 12 videos of 10 minutes each. 


required to attend class. However, class was used to work on “homework” problems which were due 
at the end of the week. The data from this item suggests that the Version 2 model was more effective 
in using the class time for homework rather than for supplemental problems. 

Students were then asked to rate their level of agreement with: “My ability to do homework prob¬ 
lems on my own was improved because of additional time spent problem solving in class.” Once 
again, students in Version 2 rated this statement significantly higher [£(100) = 2.0969, p = 0.0385]. 

In addition to class time, students in both sections were asked to rate their level of agreement 
with the following: “I prefer coming to class to listen to a lecture versus watching a lecture online.” 
An independent t-test showed that students in Version 1 had a greater average for this item [£(100) = 
4.10, p < 0.0001]. Students who were required to watch the longer videos which were recordings of 
class would rather attend a class lecture than watch online. Those who watched the shorter videos 
in Version 2 were more likely to disagree with this statement. 

Students were asked their preferences on how long the instructor should review material at the 
start of class. Previous work (Zappe, et al., 2009) recommended reviewing for shorter periods of 
time in class. This can be a challenge for faculty who are accustomed to presenting material in class. 
We found 91% students preferred using class time for problem solving rather than for reviewing 
online lectures. Furthermore, 85% of students recommended using less than 20 minutes (out of 
50 minutes) for review in class (Figure 7). 


WINTER 2015 


19 





















ADVANCES IN ENGINEERING EDUCATION 

The Evolution of a Flipped Classroom: Evidence-Based Recommendations 


Item 

Version 1 

Version 2 

p value 

I would prefer to use class time for problem solving activities, rather than 
listening to a lecture. 

3.83 

(s = 0.94) 

4.06 

(s = 0.95) 

0.2449 

I feel that moving lecture material out of class and having more time to work 
during class is a beneficial use of course time. 

3.22 

(s = 1.05) 

3.79 

(s = 1.03) 

0.0093* 

My ability to do homework problems on my own was improved because of the 
additional time spent problem solving in class. 

3.14 

(s = 1.18) 

3.58 

(s = 0.91) 

0.0385* 

I prefer coming to class to listen to a lecture versus watching a lecture online. 

3.69 

(s = 1.22) 

2.67 

(s = 1.19) 

<0.0001* 


Table 4. Students in both versions were asked to rate their level of agreement with 
the several statements using a Likert-type scale from Strongly Disagree (coded as 1) 
to Strongly Agree (coded as 5). Means and standard deviations and p value for items 
administered to students in Version 7 and Version 2. *indicates significant differences 
in the responses between Version 1 and Version 2 using a = 0.05. 


How does the type of instruction affect student learning measured with cumulative final exam scores? 

Figure 8 shows the average final exam score grades by semester. A one-way ANOVA was con¬ 
ducted on the data which showed no significant difference in average exam grades across the 
semester [F(5) = 0.255, p = 0.937]. The average grades on the final exam were surprisingly similar 
across the semesters, varying only a few points. 



0 1-10 11-20 21-30 31-40 

Preference time for review 

Figure 7. Students from Version 2 report that they would like to spend less than 20 
minutes (out of 50 minutes) of class time reviewing material from the lectures. 
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F08 F09 Sll Fll S12 F12 

Figure 8. Average Final Exam Scores for the same course with varying instructional 
strategies from Fall 2008 (F08) until Fall 2012 (F12). The grades are simitar for traditional 
lecture-based courses and both versions of the flipped course. Vertical bars represent the 
points that are one standard deviation above and below the mean. 


What aspects of the flipped format do students find most valuable? 

In general students prefer the flipped format of class to a traditional lecture format, even if they 
are a bit hesitant at first. For example in Fall 2012 a student was initially against the flipped method 
but by the end of the semester had changed his mind. 

" Initially ; / was extremely against the idea of a flipped course. To me it sounded like self- 
teaching. But after the first few weeks i realized it was not so and was surprisingly easy to 
learn from. "(Fall 2012 Post Survey) 

On the survey for Version 2, students were asked whether they would prefer to take a flipped- 
course or a traditional course with the same instructor. In the post-survey about three out of four 
(77%) of the students stated that they would prefer to take a flipped course with the instructor. 
The students were asked to explain their reasons. The major themes that emerged from students’ 
comments included: A. flexibility in learning, B. being able to re-watch the lectures and C. interact¬ 
ing with peers and faculty in order to have homework questions answered in class. An example of 
a quote including all three of these themes follows: 

7 really like watching lectures out of class. / am able to watch them at my own pace (A.) 
and can rewind when / lose focus (which / often do) (B). Also, working out problems in class 
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Code 

% of students 

Exemplar quotes 

A. Flexibility in 
learning 

38% 

“Flipping the course gave me more freedom toward how I wanted to complete 
the work.” 

B. Reviewing lectures 
& helping with focus 

35% 

The pause and rewind features of the flipped classroom.. .allow me to pay 
more attention to the lecture.” 

“If I miss something or I just want to go into a daze for a minute, I can go 
back and watch that part again.” 

C. Interaction with 
students and faculty 

33% 

“I.. .have more questions about the homework than the lectures.” 

“Coming to class ready with questions or homework lets the student interact 
with the instructor more hands-on.” 

“I found really interesting the fact of solving in-class problems since I have 
the opportunity to exchange my ideas with others and to have a different 
perspective of what is shown.” 


Table 5. Student explanations for preferring flipped course to traditionally taught 


courses. 


helps me to understand the concepts that / learned out of class better and it better prepares 
me for quizzes/exams (C.).” 

Table 5 provides example quotes illustrating the themes from this question on the survey. 

Flexibility in Learning and ability to re-watch segments 

Thirty eight percent of the students mentioned enjoying flexibility of freedom in learning. They 
mentioned being able to watch the videos “on my own time.” This flexibility benefits the student’s 
schedules as they can decide when they have the time to fit in watching the video segments. In ad¬ 
dition, 35% of students specifically mentioned preferring the fact that they can review the lectures 
on their own time. They can also pause the video and take a break when they start to lose focus 
which allows them to control their learning environment. 

During the Spring 2013, a focus group was held with students that had the same instructor for 
both the Flipped Version 2 class and a lecture-based course. One student shared: 


Y The flip method] definitely hammers down the concepts better just because I can learn on 
my own time and I can always go back within a lecture and look it up and watch it again if i 
don't understand something." (Spring 2013 Focus Group) 

In contrast a student who is taking a traditional lecture-based course with the same instructor said: 

"Sometimes i find myself just copying down whatever she writes on the board and not really 
absorbing it."(Spring 2013 Focus Group discussing live lecture) 
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Interactive class time 

Approximately one third of the students mentioned preferring the interaction with both students 
and faculty during class time on the post-survey. 

Y One advantage is] working with other students, being able to immediately have questions 
answered whether it be by a classmate, [the teaching intern], or [the instructor].” (Fall 2012 
Post survey) 

7 liked being able to ask both the professor and TA questions about the problem sets 
during the scheduled time for class because I didn’t always have the time to ask questions 
during either of their office hours. Receiving help on problems leads to less frustration and a 
better understanding of the material.” (Fall 2012 Post survey) 

During the Spring 2013 focus group students were comparing this course with a traditional 
course and discussed that the class time allowed them to get questions answered without feeling 
embarrassed. 

"And with the homework, you can do it in class with your friends, your peers, it makes it 
easier because you’re bouncing ideas off each other, it helps, it makes you think better.” 

(From Spring 2013 focus group) 

7 would much rather ask one of my friends a question, that’s why 370 [the flipped course] 
was a lot better, just because you didn’t have to ask her [the professor]. When you go to do 
your homework your friends would help you out. it’s easier when you do have a question 
because not everyone is looking at you and critiquing your question or calling you an idiot.” 
(Spring 2013 focus group) 

Opportunities for Field Trips 

In addition to working on problems in class, students really enjoyed the ability to go on field trips 
in this course. 78% of the students agreed or strongly agreed that “going on field trips increased 
my interest in Environmental Engineering.” 

"You get a feet for what you’re actually doing [while on field trips]” (Fall 2012 Focus Group) 

“The field trips are interesting, i don’t think we’ve done field trips with any other class.” (Fall 
2012 Focus Group) 

“You actually get to see things close to campus that are relevant to the class and you get to 
relate back to the class.” (Fall 2012 Focus Group) 
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" ...the field trips were very informative and interesting. They allowed us to see in reality the 
things we had been discussing in class and that PSU is a leader in many aspects of “green” 
engineering and science.” (Post survey 2012) 

The students also preferred the organization of the flipped class. 

During the Fall and Spring Focus Groups a number of students commented on the organization 
of the flipped Version 2 course - especially having the quiz the same week as the material covered. 

7 like the idea of quizzes every week, it keeps you on top of all the information. You don’t 
have to cram it in once every two months.” (Fall 2012 Focus Group) 

“This is a very well organized class...For each section [module] we are given a specific due 
date for each part - when the assessment is due, when the homework is due, when the quiz 
is taken, so we know what’s going on all the time.” (Fall2012 Focus Group) 

7 definitely like how you have to take the online assessments Sunday prior to the week of 
what would normally be lecture. So then you already have the lecture material done by 
Sunday and the whole week you can relearn it/learn it better. Just apply it.” (Spring 2013 
Focus Group) 

Finally, students see that this is the future of education 

7 have the feeling that more classes are switching to this format, or at least becoming 
fully online classes at Penn State. I’d personally rather see more classes become flipped 
classes than online classes, as / feel there’s more value to meeting in class to discuss things 
and review material and solve problems. The way online classes are currently run, from 
my experience, there really isn’t much interaction, and / don’t feel like students really get 
their money’s worth. I’ve had a couple other flipped classes before this, and they have 
contributed to my learning more than most conventional classes I’ve taken. It definitely 
seems like an ideal format for teaching, and I hope there are more like this in the future.” 

(Post survey 2012) 


DISCUSSION 

The main goal of flipping a course is to actively engage the student during class time. By mov¬ 
ing the lecture time out of the classroom, class time can be used for any number of activities that 
engage the student. In this publication, we are interested in determining students’ preferences in the 
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delivery of the flipped course. We specifically hypothesized that the refinements made, particularly 
the more customized and shorter videos and the use of in-class time to complete homework rather 
than additional problem sets, will result in more positive feedback from students. In addition we 
wanted to determine how student learning, as determined by Final Exam scores, was affected. Finally 
we wished to ascertain what specific benefits the students’ observed being in a flipped classroom. 

How do students interact with both on-line lectures and flipped classroom time? 

The majority of students watched the online videos to obtain the technical knowledge required in 
this course. The students re-watched the video content for a number of reasons including: to clear 
up a misunderstanding, when taking the online assessment, preparing for quizzes and working on 
the homework assignments. Over half of the students also admitted they re-watched the videos 
because they were distracted the first time they watched it. Comments from students indicated that 
they felt more comfortable in the flipped setting and would have been more embarrassed to raise 
their hand for clarification in a traditional lecture classroom. 

When given an option to come to class or not (in Version 2), most students still chose to come 
to class to participate in the problem solving activities. The largest reason for not coming to class 
is “being able to finish the problems on their own” or “preferring to work on their own” although 
students also mentioned “having other academic priorities” and “I didn’t feel like coming.” 

What are students’ preferences for on-line content delivery and use of class time? 

Students prefer to use class time for problem solving rather than listening to a lecture. They prefer 
to watch 10 minute video segments and to spend less than 20 minutes reviewing material from the 
lecture. In addition we hypothesize that the refinements made, particularly the more customized and 
shorter videos and the use of in-class time to complete homework rather than additional problem 
sets, will result in more positive feedback from students. Using a Likert scale, we found that students 
in Version 2 were more likely to think that moving the lecture out of class was beneficial. In addition, 
they were less likely to state that they prefer coming to class to listen to a lecture versus watching 
a lecture online. In addition students in Version 2 were more likely to state that their ability to do 
their homework was improved because of time spent in class. This indicates that the changes to 
Version 2 resulted in more positive feedback from the students. 

How does the type of instruction affect student learning as measured by cumulative final exam 
scores? 

Investigation of student grade data did not demonstrate significant differences in final exam 
scores across semesters in which the course was taught traditionally versus in the flipped format. At 
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minimum, the flipped format certainly did not “harm” students in terms of their grades. It is possible 
that the classroom flip instructional method does not impact understanding of technical content 
in the course, as measured by the final exam grade. However, grades are very complex, resulting 
from a variety of factors, some of which are external to the instructor (i.e. student motivation). Final 
exam score might not be the most appropriate measure to determine the impact of the classroom 
flip. Other measures, such as those measuring students’ perceptions of classroom climate may be 
more likely to yield significant differences between traditional and flipped offerings of the course. 
Strayer (2012) began to explore the impact of flipping a statistics course on students’ perceptions 
of climate. Exploring this in the Environmental Engineering course is planned for the next phase in 
our research, for which we are beginning to collect data. In addition, the benefit for students seeing 
the engineering concepts applied through field trips cannot be measured by a final exam. 

What do the students find more valuable about the flipped course? 

Based on the surveys and focus groups and the experiences of the instructor using the classroom 
flip for multiple, iterative semesters, we have identified four main benefits of the classroom flip. In 
Table 6, we list these four benefits and corresponding recommendations. 


Benefits of Classroom Flip Recommendation 


The on-line video segments allow students to control their 
own learning of the technical content. They can watch 
the videos when and how they want. Most of them make 
use of the ability to re-watch videos in order to clarify a 
difficult concept. 

Students benefit from using class time to work on relevant 
problem sets. The students feel more confident to learn and 
complete the required assignments when they are able to 
work with instructors and peers actively in the classroom. 

Flipping the class allows more time to incorporate real 
life applications to the course content. In this Introduction 
to Environmental Engineering course we were able to 
incorporate real life applications by using class time to go 
on relevant field trips. Most of these field trips were on or 
near the campus. 

Flipping the course increases the organization of the 
material and helps keep students on track. In Version 2, we 
incorporated weekly homework and in-class quizzes that 
corresponded with the video content. This ensured that 
students did not fall behind. 


For instructors using videos for a classroom flip, we 
recommend that these video segments are approximately 
10 minutes in length. A shorter length will make it easier 
for students to further control their learning and allow 
them to selectively review course material. This is also 
collaborated by previous research. 

We recommend that students be given time in class to 
work on relevant problems or projects that are part of the 
course requirements. 

We recommend using a part of class time to bring in real 
life applications of the course content. This could be 
through the use of field trips, guest speakers, projects or 
discussion of current events. 


We recommend that flipped courses use at least weekly 
assessments in the form of an on-line assessment before 
class time and have homework assignments and in-class 
quizzes be due at the end of the week. 


Table 6. Benefits and Recommendations for flipping a course based on our 
observations. 
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STUDY LIMITATIONS 

The study has several limitations that need to be discussed. The study relies primarily on student 
self-reports through focus groups and surveys, which may contain potential biases. Students may not 
always be forthright on certain questions (for example, student may not honest about their attendance 
behavior). Future studies should include additional types of measures, such as tracked data on online 
video watching (available through many course management systems) and actual attendance data. 
The use of final exam scores as a direct measure of student learning also has limitations, which are dis¬ 
cussed above. Another future study may consider examining the impact of the classroom flip through 
the use of external measures of student learning, such as concept inventories. The difficulty with using 
these tools is that most are not specifically aligned with the objectives of a specific course. However, 
some existing inventories have strong psychometric properties which may be appealing to use in an 
experimental or quasi-experimental design. This is especially true if an instrument is available in the 
discipline where the classroom flip is being implemented. Another limitation of this study is that we 
did not include any measure of prior performance or achievement levels of the students in the course. 

Perhaps most importantly, these recommendations are a guide based on teaching a third-year large 
environmental engineering course during a 15 week semester. The specific strategies used in this course 
may not be appropriate or feasible in other courses or at other institutions. For example, planning field 
trips during class time at other institutions may be difficult if appropriate venues are not close to campus. 

That being said, the classroom flip model can be easily adapted to many different course settings. In- 
class activities can be varied based on instructor preferences or to better suit specific course objectives. 
For example, some faculty members may choose to use an online assessment before every class period 
or use exams instead of quizzes. As another example, if a course objective emphasizes teamwork, an 
instructor may choose to use class time to have students work in small teams or on larger project-based 
learning activities. If a course objective emphasizes creativity or innovation, in-class time can be used 
for idea generation or for other steps in the creative process. Out-of-class time would focus on making 
sure students have the appropriate exposure to material that allows them to be successful in the in-class 
activity. Instructors interested in using the classroom flip should consider what in-class activities would suit 
their instructional style as well as their course objectives and adapt the model to best fit their situation. 


CONCLUSIONS 

Engineering faculty are faced with two challenges when teaching a large engineering class: Covering 
appropriate content while maintaining a classroom where students can actively interact with the techni- 
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cal material in order to better learn and apply it. Unfortunately these two challenges can seem mutually 
exclusive - class time must be used to cover content and students are left learning and applying the 
material on their own. One solution to this conflict is to have the students obtain new material at home 
through online video modules and then use class time (and faculty interaction time) to apply these 
new concepts. Our work has shown that students prefer this technique because they like having the 
flexibility to learn the new concepts on their own time and in their own way and they enjoy interacting 
with the faculty and students during class time. We believe that flipping a course can allow students the 
opportunity to become active learners in class through a variety of activities such as problem solving, 
guest speakers, idea generation, and field trips. The classroom flip model can be adapted to instructors 
to fit a variety of course settings. Instructors are encouraged to consider their own specific course, 
including course objectives, when determining the most appropriate in-class activities. 
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APPENDIX A: POST-SEMESTER REFLECTION SURVEY FOR VERSION 1 


1. There were 40 lectures posted on-line. How many did you watch? 

a. 0-10 

b. 11 - 15 

c. 16 - 20 

d. 21 - 25 

e. 26 - 30 

f. 31 - 35 

g. 36 - 40 

2. How many times during the semester did you watch a lecture video more than one time? (e.g. 
before the Final Exam or for help on homework assignments.) 

a. 0 times 

b. 1 time 

c. 2 times 

d. 3 times 

e. More than 4 times 

3. Did you review portions of the lecture that seemed unclear to you? 

(Almost always, Often, Sometimes, Rarely, Never) 

4. Did you watch the video straight through, or watch it in pieces and take breaks? 

a. Straight through 

b. Pieces 

c. Straight through, then reviewed unclear pieces 

d. All in one sitting, but I would pause and review certain sections 

e. none of these 

5. Each lecture is 50 minutes long. How long did you typically spend watching the lectures at 
one sitting? 

a. less than 10 mins, 

b. 15 mins 

c. 20 mins 

d. 30 mins 

e. 45 mins 

f. 1 Hour 

g. More than 1 hour 
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6. What length of posted lectures would you find optimum? 

a. 10 min 

b. 20 min 

c. 30 min 

d. 40 min 

e. 50 min 

f. 1 hour 

7. How many weeks did you miss at least one lecture (never watched it or watched it late) 

8. How many weeks did you miss at least one lecture (never watched it or watched it late). Number 

of weeks;_ 

9. Did you find the time spent in class problem solving helpful to your understanding of the concepts? 
(Yes, Somewhat, No) 

10. If yes or somewhat, please explain what you found most valuable. 

11. As a Professional Engineer you will be required to have 30 hours every 2 years to maintain your 
license. At this point how likely are you to obtain these hours in the following ways, (not sure, 
unlikely, somewhat unlikely, likely, very likely) 

a. On-line webinars (1-3 hour on-line lectures) 

b. Attend a certification live class 

c. Attend a technical conference 

d. Attend a university seminar 

e. Teach a technical course 

Please rate how much you agree with the following statements (Strongly Disagree, Disagree, Neither 
Agree nor Disagree, Agree, Strongly Agree) 

12. I prefer to coming to class to listen to a lecture vs. watching a lecture on-line. 

13. I would prefer to use class time for problem solving activities, rather than listening to a lecture. 

14. There were a number of times during the on-line lectures that I would have asked a question 
if I were able. 

15. It was easy to get distracted when listening to the video lecture. 

16. Too much time was dedicated to the in-class activities. 


32 


WINTER 2015 






ADVANCES IN ENGINEERING EDUCATION 

The Evolution of a Flipped Classroom: Evidence-Based Recommendations 



17. My understanding of the estimating methods was improved because of the additional time 
spent problem solving in class. 

18. I felt prepared to complete problems in class after listening to the video content. 

19. I feel that moving lecture material out of class and having more time to work in groups in practi- 
cum is a beneficial use of course time. 

20. The online quizzes were beneficial for my understanding of the course material. 

21. I would have watched the videos if we were not required to take quizzes on the material. 

How do student’s current interact with their peers outside of class? 

22. Working with other students in the smaller groups helped me 

23. I understand the concepts in the class. 

24. I often let others lead the problem solving in class. 

25. I often felt lost in my group. 

26. I was often the leader in the problem solving in the class. 

27. I thought that some of the students did not participate enough in the problem solving in class. 

28. There are 7 homework assignments in this course. For how many of these assignments did you 
work with students outside the classroom? 

a. 0 

b. 1 

c. 2 

d. 3 

e. 4 

f. 5 
g- 6 
h. 7 
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APPENDIX B: POST-SEMESTER REFLECTION SURVEY FOR VERSION 2 

1. If Dr. Velegol were teaching another course in your major both as a traditional course and as a 
flipped course, which one would you choose? 

Traditional 

Flipped 

2. Please briefly explain your reasons for this choice. 

3. For CE 370 with Dr. Velegol, which is a better use of class time? 

Reviewing Online Lectures 

Problem-Solving 

4. How long should Dr. Velegol spend on reviewing the modules in class? 

0 min, 1-10 min, 11-20 min, 21-30 min, 31-40 min, 41-50 min 

5. What other things would you have preferred to do during classtime? 

6. Given the choice, which module schedule would you prefer? 

Four - 30 minute videos 

Six - 20 minute videos 
Twelve -10 minute videos 
Twenty-four - 5 minute videos 

7. How many of the 11 modules did you watch completely? 

(0, 1-2, 3-4, 5-6, 7-8, 9-10, 11) 

8. How many of the 132 video sections did you watch more than once? 

(0, 1-10, 11-30, 31-60, 61-100, 101-132) 

9. What were your reasons for re-watching the video sections (select all that apply)? 

a. Review for quiz 

b. Help with problem sets 

c. To clear-up a misunderstanding 

d. Other 

10. We met as a group (without quizzes) about 20 times for in-class problem solving. How many 
of these class times did you miss during the semester? 

0, 1-2, 3- 5, 6-9, 10 - 15, 16-20 

11. What were your reasons for missing class? 

Was able to finish problem set on my own 
Prefer to work on the problems at home 
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Illness 

Interviews 

Didn’t feel like coming 
Other 

12. How many class times did you miss in other courses in your major this semester? 

Much less than in this class 

Less than in this class 
About the same as in this class 
More than in this class 
Much more than in this class 

13. How many times were you able to complete your homework during classtime? 

0 (I did not attend class to work on homework) 

0 (I attended class but was not able to complete my homework) 

1, 2, 3, 4, 5, 6, 7, 8, 9, 10 

Please indicate your level of agreement/disagreement with the following statements 
(Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree) 

14. I feel confident in my ability to use the material learned in this course to solve course problems. 

15. I feel confident in my ability to use the material learned in this course to solve problems in my 
future career. 

16. I feel confident in my ability to use the material learned in this course to solve problems in my 
other courses. 

17. I feel confident in my ability to use the material learned in this course to solve important prob¬ 
lems in the field. 

18. I feel more comfortable taking an online course having taken this flipped course. 

19. I feel more comfortable taking a flipped course having taken this flipped course. 

20. I feel more comfortable taking a MOOC having taken this flipped course. 

21. This course is more structurally organized than my other courses. 

22. The format of this class makes me feel more comfortable asking questions/interacting with my 
professors. 

23. The format of this class makes me feel more comfortable asking questions/interacting with 
my peers. 

24. Open Comment Box 
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APPENDIX C: FOCUS GROUP QUESTIONS FOR V2 (FALL 2012) 

(Given to students in flipped class only) 

Why are you taking this class? 

Would you take this class if you weren’t required to? Why? 

What were your goals going into this course? 

What was your primary motivation to achieve this/these goals? 

What value do you see in this course? 

Value definition: in other words, what do you feel you might be getting out of this course either 
personally or professionally? 

From a professional standpoint, which part of this course has the most value (ie you’ll be most likely 
to use in the future)? 

Was this what you initially thought? 

How has this course changed your opinion on Environmental Engineering, if at all? 

What other CE classes have you taken/are you currently taking? 

Describe the structure of this/these courses. 

What aspect of 370’s flipped structure has been most beneficial to your learning in comparison 
to this other course? 

What aspect of 370’s flipped structure has been least beneficial to your learning in comparison 
to this other course? 

Pretend this course is taught in the same format by another professor. If students mention teacher 
attitude. 

How would you describe the classroom environment in CE 370? 

For example, how do you feel that the instructor interacts with students in the class? 

Do you feel that the environment is support or unsupportive to your learning? 

Describe why you feel this way. 

Open Forum ending 
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APPENDIX D: FOCUS GROUP QUESTIONS FOR V2 (SPRING 2013) 

(Given to 4 students who had the same instructor for both a flipped class and a traditional 
lecture-based course.) 

Why are you taking this class? 

Would you take this class if you weren’t required to? Why? 

What were your goals going into this course? 

What was your primary motivation to achieve this/these goals? 

What value do you see in this course? 

Value definition: in other words, what do you feel you might be getting out of this course either 
personally or professionally? 

From a professional standpoint, which of these courses has the most value? Why? 

In which course do you feel more able to do well? Why? 

What aspect of 370’s flipped structure has been most beneficial to your learning in comparison to 371? 

What aspect of 370’s flipped structure has been least beneficial to your learning in comparison to 371? 

How would you describe the classroom environment in CE 371 compared to CE 370? For example, 
how do you feel that the instructor interacts with students in the class? 

Do you feel that the environment is support or unsupportive to your learning? 

Describe why you feel this way. 

Open Forum ending 
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